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DETAILED ACTION 

Response to Amendment 

1 . The amendment filed on 06/06/2008 has been entered. The amendments to the 
claims overcomes the 35 USC 1 12 first paragraph rejection set forth in the office action 
mailed on 03/06/2008. 

Response to Arguments 

2. Applicant's arguments filed 06/06/2008 have been fully considered but they are 
not persuasive. 

The arguments concerning the new matter objection to the specification is not 
persuasive The amendment filed 01/07/2008 is objected to under 35 U.S.C. 132(a) 
because it introduces new matter into the disclosure. 35 U.S.C. 132(a) states that no 
amendment shall introduce new matter into the disclosure of the invention. The added 
material which is not supported by the original disclosure is as follows: Changing "real 
object" to "operating unit" is not supported by applicants originally filed specification. 

The "operating unit" language added to the specification corresponds to 
"operating unit 1090" of applicants specification which does not measure position and 
orientation of a plurality of positions in real space "for setting a constraining shape by 
using a shape generated based on the inputted three-dimensional position information 
in case of the input of the constraining shape" (01/07/2008 specification amendment). 
Applicants specification in paragraphs 30, 31, 32, 44, 69, 82, 83, 84, and 85 discuss 
operation unit 1090 as controlling the virtual object and in paragraphs 28, 30, 42, and 50 
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discuss stylus 1060 as setting the position and orientation of a plurality of positions in 
real space used "for setting a constraining shape by using a shape generated based on 
the inputted three-dimensional position information in case of the input of the 
constraining shape". The specification does not convey the operating unit 1090 is 
moved to measure the three-dimensional position information used "for setting a 
constraining shape by using a shape generated based on the inputted three- 
dimensional position information in case of the input of the constraining shape" 
(01/07/2008 specification amendment). 

Applicant is required to cancel the new matter in the reply to this Office Action. 

The argument concerning Kitamura is not persuasive because an estimation of 
the 6DOF position and orientation would have been obvious to one of ordinary skill in 
the art when the 6DOF is used to take an "image" of the real world object because the 
digital representation of the position and orientation of the points of the image sensed 
by the 6DOF are an estimation of the actual points and orientation of the points of the 
image sensed by the 6DOF. Thus, when one of ordinary skill in the art uses the 6DOF 
to take an "image" of the real world object than an estimation of the position and 
orientation of the image capturing means or unit and setting of the constraining shape is 
performed because this will allow the user to "bring a real object into a virtual world", 
page 133 column 1 lines 18-23 of the abstract and page 134 column 1 last sentence of 
the first paragraph of section 2. Also note the discussion of using the manipulator, 
6DOF, to form a constraining plane at page 136 column 1 lines 8-1 1 and figure 2. 
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With regard to the amendment to claims 3 and 18 note Kitamura at page 137 
column 1 lines 1-3. 

Thus, the new matter rejection is maintained and reproduced below and the 35 
USC 103 rejection based upon Kitamura is maintained and set forth below with 
modification to reflect claim amendments. 

Claim Rejections - 35 USC §112 

3. The following is a quotation of the second paragraph of 35 U.S.C. 1 12: 

The specification shall conclude with one or more claims particularly pointing out and distinctly 
claiming the subject matter which the applicant regards as his invention. 

4. Claims 10-15 are rejected under 35 U.S.C. 112, second paragraph, as being 
indefinite for failing to particularly point out and distinctly claim the subject matter which 
applicant regards as the invention. 

Claim 10: 

At line 4 "estimating the position and orientation" is claimed and at line 8 
"measuring the position and orientation" is claimed and at lines 12 and 13 position and 
orientation information is used but are unclear which position and orientation 
information, "estimated" or "measured", is used. Also unclear is whether the "estimated" 
and "measured" position and orientation information are the same or different position 
and orientation information. 
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Claim Rejections - 35 USC §103 

5. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 102 of this title, if the differences between the subject matter sought to be patented and 
the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 



6. The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1 , 148 
USPQ 459 (1966), that are applied for establishing a background for determining 
obviousness under 35 U.S.C. 103(a) are summarized as follows: 

1 . Determining the scope and contents of the prior art. 

2. Ascertaining the differences between the prior art and the claims at issue. 

3. Resolving the level of ordinary skill in the pertinent art. 

4. Considering objective evidence present in the application indicating 
obviousness or nonobviousness. 

7. This application currently names joint inventors. In considering patentability of 
the claims under 35 U.S.C. 103(a), the examiner presumes that the subject matter of 
the various claims was commonly owned at the time any inventions covered therein 
were made absent any evidence to the contrary. Applicant is advised of the obligation 
under 37 CFR 1 .56 to point out the inventor and invention dates of each claim that was 
not commonly owned at the time a later invention was made in order for the examiner to 
consider the applicability of 35 U.S.C. 103(c) and potential 35 U.S.C. 102(e), (f) or (g) 
prior art under 35 U.S.C. 103(a). 
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8. Claims 1-3 and 5-18 are rejected under 35 U.S.C. 103(a) as being unpatentable 
over Yoshifumi Kitamura and Fumio Kishino, Consolidated Manipulation of Virtual and 
Real Objects, September 1997, Proceedings of the ACM symposium on Virtual reality 
software and technology, pages 133-138. Kitamura teaches an augmented reality 
system that uses object constraints to control the visual interaction between the virtual 
objects and the real objects. 

A detailed analysis of the claims follows. 

Claim 10: 

Kitamura teaches an information processing method for changing the position 
and orientation of a virtual object in mixed reality space obtained by combining a real 
space and a virtual space, said method comprising the steps of: 

Kitamura does not fully teach: 

an estimating step of estimating position and orientation of an image capturing 
means (An estimation of the 6DOF position and orientation would have been obvious to 
one of ordinary skill in the art at the time of the invention when the 6DOF is used to take 
an "image" of the real world object as discussed below because the digital 
representation of the position and orientation of the points of the image sensed by the 
6DOF are an estimation of the actual points and orientation of the points of the image 
sensed by the 6DOF. Thus, when one of ordinary skill in the art uses the 6DOF to take 
an "image" of the real world object as discussed below than an estimation of the 
position and orientation of the image capturing means or unit and setting of the 
constraining shape is performed because this will allow the user to "bring a real object 
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into a virtual world", page 133 column 1 lines 18-23 of the abstract and page 134 
column 1 last sentence of the first paragraph of section 2. Also note the discussion of 
using the manipulator, 6DOF, to form a constraining plane at page 136 column 1 lines 
8-11 and figure 2.); 

Kitamura further teaches: 

obtaining three dimensional position information of a plurality of positions 
designated by an input unit moved by a user in the real space {In view of page 135 first 
paragraph under the heading of 5 Consolidated Manipulation Environment of Kitamura 
the same 6 DOF tracker device is used to control the position of a virtual world object(s) 
and to control the position of a constraining real world object(s) which is similar to 
applicants system where stylus 1060 is used to control the virtual world object(s) and to 
control the location of the constraining real world object(s), see applicants paragraphs 
[0028], [0030], and [0041]. The 6 DOF has a stylus the user moves which meets the 
"operating unit moved by a user in the real space" claim limitation.), the input unit being 
capable of measuring the position and orientation in the real space {The 6 DOF 
measures position and orientation in real space of the sylus on the 6 DOF.); 

determining an input of a constraining shape or an operation of the virtual object 
{The system determines if a shape having constraints is being input for the real object 
such as a toy block (section 4.1) or surface (section 5.4) or such as moving the real 
object which has a constraining shape and the system determines if the user is moving 
the virtual object); 
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Kitamura does not fully teach: 

obtaining a constraining shape by using a shape generated based on the 
obtained three-dimensional position information in the case of the input of the 
constraining shape (The first paragraph in section 5 on page 135 describes the user 
using a 6 DOF tracker device to manipulate the virtual world objects and to position the 
constraining real world objects. The shape of the real world object(s) are used to 
constrain the movement of the virtual world object(s) by giving the real world object(s) a 
shape that the virtual world object(s) interacts with in a constrained manner. A step of 
inputting the shape of the real world objects constraining shape is inherently present). 
Kitamura's use of the 6 DOF tracker device to control the position of the real world 
object suggests using the same 6 DOF tracker device to input the constraining shape of 
the real world object since this would require less input devices for the user to use and 
learn how to use. Additionally section 2 at lines 7-14 states "To bring an object that 
already exist in the real world into a computer-generated virtual world, it is necessary to 
construct accurate shape representation of the real object in a computer system. A 
traditional method for this is to use conventional modeling software after precisely 
measuring the size or length of the real object by hand." which suggests using a 
computer input device to measure the real object in order to have accurate shape 
representation of the real object's constraining shape. Thus, it would have been 
obvious to one of ordinary skill in the art at the time of applicants invention to input with 
the 6 DOF tracker device the 3D coordinates of the constraining shape because the toy 
block (section 4.1) or surface (section 5.4) need to have their respective constraining 
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coordinates input in order for the virtual object to properly interact with the real objects 
since this will require less input devices for the user to use and learn how to use and 
because section 2 at lines 7-14 suggests using a computer input device to measure the 
real object in order to have accurate shape representation of the real object's 
constraining shape. To one of ordinary skill in the art the position and orientation of the 
real objects that will be interacted with the 6 DOF tracker would be efficiently known in 
advance by using the 6 DOF tracker to measure the real object by the user moving the 
6 DOF tracker "by hand". KSR International Co. v. Teleflex Inc., 82 USPQ2d 1385 (U.S. 
2007), U.S. Supreme Court No. 04-1350 Decided April 30, 2007, 127 SCt 1727, 167 
LEd2d 705.; 

Kitamura further teaches: 

changing the position and orientation of the virtual object according to 
instructions from the user, based on the obtained constraining shape as constraint 
condition in case of the operation of the virtual object {The first paragraph in section 5 
on page 135 describes the user using a 6 DOF tracker device to manipulate the virtual 
world object(s). The shape of the real world object(s) are used to constrain the 
movement of the virtual world object(s) by giving the real world object(s) a shape that 
the virtual world object(s) interacts with in a constrained manner. Sections 5. 1 to 5.4 
discusses manipulation of the virtual world object based upon constraint conditions 
based on the shape of the real world object in response to the user using the 6 DOF 
tracker device. The constraining shape generated from the three-dimensional position 
information constrains the interaction of the virtual world object with the real world 
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object, see sections 2, 4 to 5.4. The introduction on page 133 second full paragraph 
discusses augmented reality which synthesizes a virtual object with a real object. 
Section 5. 1 discusses after movement is detected by the 6DOF manipulator the virtual 
object is moved according to the constraints.); and 

combining an image of the virtual object generated according to the changed 
position and orientation, and the real image (The introduction on page 133 second full 
paragraph discusses augmented reality which synthesizes a virtual world object with a 
real world object. Section 5. 1 discusses after movement is detected by the 6DOF 
manipulator the virtual world object is moved according to the constraints of the real 
world object. Sections 2, 4, and 5 discusses mixing the virtual and real world images of 
virtual and real world objects.). 

Claim 11: 

Kitamura teaches an information processing method according to Claim 10, 
further comprising the step of combining a virtual image indicating the constraining 
shape with the real image (The introduction on page 133 second full paragraph 
discusses augmented reality which synthesizes a virtual object with a real object. 
Section 5. 1 discusses after movement is detected by the 6DOF manipulator the virtual 
object is moved according to the constraints. Sections 2, 4, and 5 discusses mixing the 
virtual and real world images of virtual and real world objects.). 

Claim 12: 

Kitamura teaches an information processing method according to Claim 10, 
wherein the constraining shape is a plane. (On page 136 in the text above figure 2 
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discusses determining a plane and using the plane to constrain movement of the virtual 
object is discussed with regards to figure 2.). 
Claim 13: 

Kitamura teaches an information processing method according to Claim 10, 
wherein said changing step includes changing the position and orientation of the virtual 
object by changing the position and orientation of the operating unit {The 6 DOF tracker 
device is an operating unit. The user using the 6 DOF tracker device manipulates the 
virtual objects by changing the position and orientation of the 6 DOF tracker device.). 

Claim 14: 

Kitamura teaches a computer program product comprising a computer readable 
medium storing computer program code for performing the information processing 
method according to Claim 10 wherein the information processing method is executed 
by a computer device (This article is directed to computers that generate the augmented 
reality scene since it was published by ACM for a symposium on virtual reality software 
and technology and since at page 133 in the last sentence in the second paragraph of 
section 1 "software/hardware" is discussed. Software causing a computer to perform 
Kitamura's augmented reality is inherently stored in a computer readable recording 
medium.). 

Claim 15: 

Kitamura teaches a computer-readable recording medium, storing the computer 
program according to Claim 14 (Software causing a computer to perform Kitamura's 
augmented reality is inherently stored in a computer readable recording medium.). 
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Claim 16: 

The functions of Kitamura corresponds to the claimed units because the software 
causes the computer to become a unit that performs a process. As seen below 
Kitamura performs the claimed processes. 

Kitamura teaches an information processing device for aiding control operations 
relating to controlling the position and orientation of a virtual object, said device 
comprising: 

an image capturing unit configured to capture a real image in real space {The 
measurements by the user in three dimensional real space, the device for obtaining the 
3-D shape by using a range image, and the device for obtaining the 3-D shape by using 
multiple cameras captures a real image in real space. The three paragraphs found in 
section 2 on page 134 of Kitamura teaches the user measuring the real objects, a 
device measuring the real objects with range in response to the user, or a device 
measuring the real objects with multiple cameras range in response to the user. The 
claim does not limit the form of the real image, thus, Kitamura teaches this limitation. 
Note applicants paragraph [0024] does not restrict the image capturing unit to a 
particular type); 

Kitamura does not fully teach: 

an estimating unit for estimating the position and orientation of said image 
capturing unit (An estimation of the 6DOF position and orientation would have been 
obvious to one of ordinary skill in the art in the art at the time of the invention when the 
6DOF is used to take an "image" of the real world object as discussed below because 
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the digital representation of the position and orientation of the points of the image 
sensed by the 6DOF are an estimation of the actual points and orientation of the points 
of the image sensed by the 6DOF. Thus when one of ordinary skill in the art uses the 
6DOF to take an "image" of the real world object as discussed below than an estimation 
of the position and orientation of the image capturing means or unit and setting of the 
constraining shape is performed because this will allow the user to "bring a real object 
into a virtual world", page 133 column 1 lines 18-23 of the abstract and page 134 
column 1 last sentence of the first paragraph of section 2. Also note the discussion of 
using the manipulator, 6DOF, to form a constraining plane at page 136 column 1 lines 
8-11 and figure 2.); 

Kitamura further teaches 

a virtual image generation unit configured to generate a virtual image of a virtual 
object according to the position and orientation of said image capturing unit (Kitamura 
discusses in section 2 in the first paragraph using conventional modeling software after 
precisely measuring the size or length of the real object by hand or by the devices which 
is an generation unit capable of generating three-dimensional positional information. 
Sections 2, 4 and 5 discuss generating a virtual image(s) of a virtual object(s).) ; 

a superimposing unit configured to superimpose the generated virtual image with 
the captured real image (Sections 2, 4, and 5 discusses mixing the virtual and real world 
images of virtual and real world objects.); 

a determination unit configured to determine an input of a constraining shape or 
an operation of the virtual object (The system determines if a shape having constraints 
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is being input for the real object such as a toy block (section 4.1) or surface (section 5.4) 
or such as moving the real object which has a constraining shape and the system 
determines if the user is moving the virtual object); 

an inputting unit configured to input three-dimensional position information of a 
plurality of positions inputted by a moving a operating unit in the real space by a user (In 
view of page 135 first paragraph under the heading of 5 Consolidated Manipulation 
Environment ofKitamura the same 6 DOF tracker device is used to control the position 
of a virtual world object(s) and to control the position of a constraining real world 
object(s) which is similar to applicants system where stylus 1060 is used to control the 
virtual world object(s) and to control the location of the constraining real world object(s), 
see applicants paragraphs [0028], [0030], and [0041]. The 6 DOF has a stylus the user 
moves which meets the "moving a real object limitation in the real space by a user" 
claim limitation.), the operating unit being capable of measuring the position and 
orientation in the real space (The 6 DOF measures position and orientation in real 
space of the sylus on the 6 DOF.); 

Kitamura does not fully teach: 

a setting unit configured to set a constraining shape by using a shape generated 
based on the inputting three-dimensional position information in the case of the input of 
the constraining shape (The first paragraph in section 5 on page 135 describes the user 
using a 6 DOF tracker device to manipulate the virtual world objects and to position the 
constraining real world objects. The shape of the real world object(s) are used to 
constrain the movement of the virtual world object(s) by giving the real world object(s) a 
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shape that the virtual world object(s) interacts with in a constrained manner. A step of 
inputting the shape of the real world objects constraining shape is inherently present). 
Kitamura's use of the 6 DOF tracker device to control the position of the real world 
object suggests using the same 6 DOF tracker device to input the constraining shape of 
the real world object since this would require less input devices for the user to use and 
learn how to use. Additionally section 2 at lines 7-14 states "To bring an object that 
already exist in the real world into a computer-generated virtual world, it is necessary to 
construct accurate shape representation of the real object in a computer system. A 
traditional method for this is to use conventional modeling software after precisely 
measuring the size or length of the real object by hand." which suggests using a 
computer input device to measure the real object in order to have accurate shape 
representation of the real object's constraining shape. Thus, it would have been 
obvious to one of ordinary skill in the art at the time of applicants invention to input with 
the 6 DOF tracker device the 3D coordinates of the constraining shape because the toy 
block (section 4.1 ) or surface (section 5.4) need to have their respective constraining 
coordinates input in order for the virtual object to properly interact with the real objects 
since this will require less input devices for the user to use and learn how to use and 
because section 2 at lines 7-14 suggests using a computer input device to measure the 
real object in order to have accurate shape representation of the real object's 
constraining shape. To one of ordinary skill in the art the position and orientation of the 
real objects that will be interacted with the 6 DOF tracker would be efficiently known in 
advance by using the 6 DOF tracker to measure the real object by the user moving the 
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6 DOF tracker "by hand". KSR International Co. v. Teleflex Inc., 82 USPQ2d 1385 (U.S. 
2007), U.S. Supreme Court No. 04-1350 Decided April 30, 2007, 127 SCt 1727, 167 
LEd2d 705.; and 

Kitamura further teaches: 

an operating unit configured to control the position and orientation of the virtual 
object based on the constraining shape in accordance with the a user's instruction in 
case of the operation of the virtual object.( Sections 5.1 to 5.4 discusses manipulation of 
the virtual world object based upon constraint conditions based on the shape of the real 
world object in response to the user using the 6 DOF tracker device. The constraining 
shape generated from the three-dimensional position information constrains the 
interaction of the virtual world object with the real world object, see sections 2, 4 to 5.4. 
The introduction on page 133 second full paragraph discusses augmented reality which 
synthesizes a virtual object with a real object. Section 5. 1 discusses after movement is 
detected by the 6DOF manipulator the virtual object is moved according to the 
constraints.). 

Claim 17: 

Kitamura teaches an information processing device according to Claim 16, 
wherein the constraining shape is defined by polygons {Figure 2's Cubes are formed 
with polygons.) and the apexes of the polygons are at positions inputted by the user (As 
discussed in the a setting unit above it would have been obvious to use the 6 DOF 
tracker device to input the constraining shape of the real world object which would input 
the apexes of the constraining shape of the polygon.) or the constraining shape is a 
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plane passing through the positions inputted by the user (A real sensed surface is a 
planar real world object whose position is inputted by the user in the setting unit. As 
discussed in the a setting unit above it would have been obvious to use the 6 DOF 
tracker device to input the constraining shape of the real world object which would input 
plane for a surface.). 
Claim 18: 

Kitamura teaches an information processing device according to Claim 16, 
wherein said operating unit performs at least one of the following operations in 
performing an operation controlling the position and orientation of the virtual object: 

a translation operation for causing translational movement of the virtual object 
based on the constraining shape (Page 137 column 1 lines 1-3 teaches translation 
when the virtual object collides with the real surface.); and/or 

a rotation operation for rotating the virtual object on an axis which is a normal 
vector at a plane where the constraining shape and the virtual object come into contact 
{Page 137 column 1 lines 1-3 teaches rotation when the virtual object collides with the 
real surface.) {Page 137 column 1 lines 1-3 also teaches translation and rotation when 
the virtual object collides with the real surface). (For the added "and/" note page 137 
column 1 lines 1-3 teaches translation and rotation.) 

Claims 1-3: 

Means plus function claims 1-3 correspond to device claims 16-18 and the 
means of Kitamura, software and computer, are equivalent to applicant's means of 
software and computer. 
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Claims 5-7: 

Step for claims 5-7 correspond to device claims 16-18 and the steps of Kitamura, 
software and computer, are equivalent to applicant's steps performed by software and 
computer. 

Claims 8 and 9: 

Claims 8 and 9 mirror claims 14 and 15 addressed above and they are rejected 
for the same reasons given above for claims 14 and 15. 

Conclusion 

9. THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time 
policy as set forth in 37 CFR 1 .136(a). 

A shortened statutory period for reply to this final action is set to expire THREE 
MONTHS from the mailing date of this action. In the event a first reply is filed within 
TWO MONTHS of the mailing date of this final action and the advisory action is not 
mailed until after the end of the THREE-MONTH shortened statutory period, then the 
shortened statutory period will expire on the date the advisory action is mailed, and any 
extension fee pursuant to 37 CFR 1 .136(a) will be calculated from the mailing date of 
the advisory action. In no event, however, will the statutory period for reply expire later 
than SIX MONTHS from the mailing date of this final action. 
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Status information for unpublished applications is available through Private PAIR only. 
For more information about the PAIR system, see http://pair-direct.uspto.gov. Should 
you have questions on access to the Private PAIR system, contact the Electronic 
Business Center (EBC) at 866-217-9197 (toll-free). 
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